In order to improve the discriminative ability of dictionary, a dictionary learning algorithm based on Fisher discriminative criterion constraint of the atoms was proposed, which was called Fisher Discriminative Dictionary Learning of Atoms (AFDDL). Firstly, the specific class dictionary learning algorithm was used to assign a class label to each atom, and the scatter matrices of within-class atoms and between-class atoms were calculated. Then, the difference between within-class scatter matrix and between-class scatter matrix was taken as the Fisher discriminative criterion constraint to maximize the differences of between-class atoms. The difference between the same class atoms was minimized when the autocorrelation was reduced, which made the same class atoms reconstruct one type of samples as much as possible and improved the discriminative ability of dictionary. The experiments were carried out on the AR face database, FERET face database, LFW face database and the USPS handwriting database. The experimental results show that, on the four image databases, the proposed algorithm has higher recognition rate and less training time compared with the Label Consistent K-means-based Singular Value Decomposition (LC-KSVD) algorithm, Locality Constrained and Label Embedding Dictionary Learning (LCLE-DL) algorithm, Support Vector Guided Dictionary Learning (SVGDL) algorithm, and Fisher Discriminative Dictionary Learning (FDDL) algorithm. And on the four image databases, the proposed algorithm has higher recognition rate compared with Sparse Representation based Classification (SRC) and Collaborative Representation based Classification (CRC).